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The findings from this review do not reflect the full body of research evidence on college coaching. 


What is this study about? 

The study examined whether providing college 
application coaching to high school seniors increased 
postsecondary enrollment. The program was aimed 
at students whom school counselors believed were 
on the verge of not applying to college, despite hav- 
ing tenth-grade test scores that were high enough to 
warrant applying. 

Study authors chose high schools in New Hampshire 
based on their interest in the intervention and their 
willingness to participate in a randomized experi- 
ment. Approximately 1 ,150 students were randomly 
assigned into one of two groups. Students in the 
intervention group received the college coaching pro- 
gram, which was implemented by college students 
and provided in-person assistance with completing 
college application and financial aid forms, money 
to cover application fees, and a $100 cash incentive 
to participants for completing the college application 
process. Students in the comparison group did not 
receive the intervention (but they were eligible for the 
usual services normally available to them). 

The authors used data from the New Hampshire 
Department of Education’s Data Warehouse and 
the National Student Clearinghouse. The primary 
outcome was whether students enrolled in college, 
and follow-up analyses examined whether students 
attended at least 3 semesters of college. 


Features of the College Coaching Program 


College coaching was targeted at high school 
seniors whom school counselors identified as good 
candidates for college but who had not completed 
the college application process. It was implemented 
in the spring semester. 

The college coaching program had three main 
components: mentoring, financial support, and a 
completion incentive. Mentoring involved weekly 
meetings with college undergraduates who helped 
students identify all of the steps that needed to be 
completed and who tracked progress in these tasks, 
assisted with entrance essay development, and 
helped students fill out all applications (including 
those for financial aid). Financial support included 
funds to cover college application and entrance 
exam fees. The completion incentive was $100, and 
was given to the students after they had completed 
the application process. 

Meetings took place on school grounds, usually in a 
library or computer center, and usually with school 
guidance counselors present. 
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What did the study find? 

The authors found, and the WWC confirmed, a 
statistically significant effect of the intervention 
on postsecondary enrollment: Students who were 
offered the chance to participate in the coaching 
program were more likely to enroll in a postsecond- 
ary institution than students who were not offered 
the chance to participate (57% vs. 52%). However, 
there was a statistically significant interaction with 
student gender, such that the intervention improved 
the postsecondary enrollment rate for women (63% 
in the intervention group vs. 50% in the comparison 
group), but for men, enrollment rates were virtually 
identical across the two groups (about 53%). The 
authors found that the intervention effects on post- 
secondary enrollment were similar for both non- 
White students and White students, and for students 
eligible for free or reduced-price lunch and students 
not eligible for this program. 

For postsecondary enrollment in 3 or more semesters 
after high school graduation, the authors did not find 
a statistically significant difference between the inter- 
vention and comparison groups overall. However, as 
with the analysis of postsecondary enrollment, there 
was a statistically significant interaction with gender. 
The intervention increased the likelihood that women 
were enrolled in 3 or more semesters of college after 
high school (37% vs. 28%), but for men, the rates 
were virtually identical, at 44% across both groups. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards without reservations 

Strengths: The study is a well-implemented 
randomized controlled trial. 
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Appendix A: Study details 

Carrell, S., & Sacerdote, B. (2013). Late interventions matter too: The case of college coaching New 
Hampshire (NBER Working Paper 19031). Cambridge, MA: National Bureau of Economic Research. 
Retrieved from http://www.nber.org 


Setting 

The study was conducted in 12 high schools in New Hampshire. Schools were selected based 


on their willingness to participate in a randomized experiment. 

Study sample 

About 1 ,150 students participated in the study. The program targeted students who were 
considered to be good candidates for postsecondary attendance but whom school counselors 
believed were on the verge of not applying. Study authors randomly assigned the students 
into the intervention group and the comparison group. Overall, 55% of the sample was male, 

31 % of the students were eligible for free or reduced-price lunch, and 84% of the students 
were White. Approximately 53% of students in the sample had a tenth-grade math test score 
that was above the state median. For the state reading test, this figure was about 36%. 

Intervention 

group 

Students in the intervention group received college coaching, which was targeted at high 
school seniors whom school counselors identified as good candidates for college but who had 
not completed the college application process. The college coaching program had three main 
components: mentoring, financial support, and a completion incentive. Mentoring involved 
weekly meetings with college undergraduates who helped students identify all of the steps 
that needed to be completed and who tracked progress in these tasks, assisted with entrance 
essay development, and helped students fill out all applications (including those for financial 
aid). Financial support included funds to cover college application and entrance exam fees. 
The completion incentive was $100, and was given to the students after they had completed 
the application process. Meetings took place on school grounds, usually in a library or com- 
puter center, and usually with school guidance counselors present. 

Comparison 

group 

Students in the comparison group did not receive the intervention, but were eligible to receive 
the guidance services usually available to them. 

Outcomes and 
measurement 

The authors used data from the New Hampshire Department of Education’s Data Warehouse and the 
National Student Clearinghouse. The following outcomes were reported: (a) postsecondary enroll- 
ment, and (b) enrolled in any type of postsecondary institution, 3 or more semesters after graduation. 
For a more detailed description of these outcome measures, see Appendix B. 2 

Support for 
implementation 

College coaching was implemented by undergraduate students working full time (i.e., 20 hours 
a week) for about 10 weeks (January through mid-March). The coaching sessions took place 
on school grounds, usually in the library or computer center, and were usually attended by 
school guidance counselors. 

Reason for 
review 

This study was identified for review by the WWC because it received significant media attention. 
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Appendix B: Outcome measures for each domain 


Enrollment 


Enrollment Postsecondary enrollment information was obtained from the National Student Clearinghouse. 


Persistence 


Enrolled 3 or more semesters Postsecondary enrollment information was obtained from the National Student Clearinghouse. 

after high school graduation 

Table Notes: The study also provided results for persistence 2 years post high school graduation. This outcome is conceptually similar to enrollment in 3+ semesters after high 
school graduation, and the results observed were virtually identical across the two analyses. The study also provided results for attendance at a 4-year college (vs. attending a 
2-year college and not attending postsecondary, combined). This is not an eligible outcome as specified in the WWC Postsecondary Education Review Protocol v. 2. 

The data for Enrolled 3 or more semesters after high school graduation are from the 2009, 201 0, and 201 1 cohorts, because an insufficient amount of time had elapsed at the time of 
report publication (May 201 3) for the 201 2 cohort to be included in this analysis. 
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Appendix C: Study findings for each domain 





Mean 

(standard deviation) 

WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention Comparison 
group group 

Mean Effect Improvement 

difference size index 

p-value 


Enrollment (end of coaching program) 

Postsecondary enrollment Full sample 1,149 57% 

students 

52% 

5% 

0.13 

+5 

0.03 

Domain average for enrollment (end of coaching program) 



0.13 

+5 

Statistically 

significant 

Persistence (2 years after high school) 

Enrolled 3 or more Full sample 950 41% 

semesters after high school students 

37% 

4% 

0.10 

+4 

0.12 

Domain average for persistence (2 years after high school) 



0.10 

+4 

Not 

statistically 

significant 


Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 
a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student’s percen- 
tile rank that can be expected if the student is given the intervention. The WWC-computed average effect size is a simple average rounded to two decimal places; the average 
improvement index is calculated from the average effect size. 

Study Notes: Corrections for multiple comparisons were not needed. The p-values presented here were computed by the WWC. The sample sizes and percentages reported in this 
table are from Table 1 (p. 26) and Appendix Table 3 (p. 43) of the manuscript. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Enrollment (end of coaching program) 

Postsecondary enrollment 

Men 

632 

students 

53% 

53% 

0% 

0.00 

0 

0.95 

Postsecondary enrollment 

Women 

517 

students 

63% 

50% 

13% 

0.31 

+12 

<0.001 

Persistence (2 years after high school) 

Enrolled 3 or more 
semesters after high school 

Men 

531 

students 

44% 

44% 

0% 

-0.01 

0 

0.91 

Enrolled 3 or more 
semesters after high school 

Women 

419 

students 

37% 

28% 

9% 

0.25 

+10 

0.01 

Table Notes: Positive results for mean difference, effect size, and improvement index favor the intervention group; negative results favor the comparison group. The effect size is 


a standardized measure of the effect of an intervention on student outcomes, representing the change (measured in standard deviations) in an average student’s outcome that can 
be expected if the student is given the intervention. The improvement index is an alternate presentation of the effect size, reflecting the change in an average student’s percentile 
rank that can be expected if the student is given the intervention. 

Study Notes: The percentages reported in this table were taken from the manuscript (Table 1, p. 26, and Appendix Table 3, p. 43). The sample sizes were provided by the study 
authors. The p-values presented here were computed by the WWC. Corrections for multiple comparisons were needed but would have not changed the statistical conclusions, and 
as such, these were not applied. 

The study report provides several other subgroup analyses, based only on the women in the sample: first generation college students, racial/ethnic minorities, and free/reduced- 
price lunch eligible students. The authors reported no statistically significant differential effects on any of these subgroups, suggesting that the coaching program effects are 
similar across different categories of these groups (that is, the analyses suggest similar effect sizes for first generation college students and those who are not first generation 
college students, etc.). Results obtained from the study authors confirm these analyses. In addition, the study reports an analysis comparing students above the 75th percentile vs. 
those below the 75th percentile on a state achievement test. This is not a protocol-specified subgroup, and therefore, is not included here. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC’s assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the Postsecondary Education topic area review protocol, version 2.0. A quick review of 
this study was released on January 16, 2013, and this report is the follow-up review that replaces that initial assessment. 

2 Study authors also measured enrollment in a 4-year college (vs. a 2-year college and no enrollment) and enrollment in a 2-year college 
(vs. a 4-year college and no enrollment). However, these outcomes are not eligible for review under the Postsecondary Education topic 
area review protocol, version 2.0.) 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2013, October). 1/1 /l/l/C 
review of the report: Late interventions matter too: The case of college coaching New Hampshire. Retrieved 
from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 


October 2013 


Page 8 


